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ABSTRACT 

In 1997, the Wake County Public Schools, North Carolina, 
began a 3-year community-wide collaboration between existing programs and 
agencies expressly intended to coordinate efforts to serve at-risk children 
and their families. The Safe Schools Healthy Students grant program was built 
on 17 strategies aimed at addressing risk. Strategy 17 called for expanding 
the use of the NovaNET online learning system to all 17 high school labs and 
2 alternative middle schools. NovaNET offers a comprehensive range of high 
school curricula, and each curricular unit is aligned with North Carolina's 
Standard Course of Study. Each lesson requires students to read instructional 
material on the computer, answer diagnostic questions, and sometimes complete 
written responses. An evaluation of the NovaNET program and its use was 
conducted through focus groups, 1 with 13 lab coordinators and the other with 
13 guidance counselors, individual interviews with 10 lab coordinators, and 
interviews with 2 counselors and 25 students from 4 schools. Surveys were 
completed by 9 lab coordinators, 9 guidance counselors, 79 students, and 18 
teachers. Findings show that NovaNET is used creatively and with encouraging 
impact. More than twice the projected number of students participated in 
NovaNET, and 73.8% of attempted credits were successfully completed. Some 843 
students were back on track for graduation after being off-track before 
NovaNET use. Teachers and students saw many advantages to the system. Some 
suggestions are made for program improvement. Seven appendixes contain the 
participant surveys. (SLD) 
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Introduction 



Wake County Public Schools have addressed the needs of at-risk students with an array of 
academic and alternative programs that reflect serious commitment of funds and time. The 1999-2000 
academic year witnessed the beginning of a 9 million dollar, three-year, community-wide collaboration 
between existing programs and agencies expressly intended to coordinate efforts to serve at-risk children 
and their families. The grant, entitled Safe Schools Healthy Students , was funded by the Department of 
Justice, the Department of Health and Human Services, and the Department of Education in an effort to 
promote safe, inviting, and optimal learning environments for all students. The program was built upon a 
broad collection of 17 strategies aimed at addressing the risk and resilience factors associated with school 
safety and student success. Each strategy included a three-year sequence of action steps, focused on 
strengthening resources or services within the school system. These services ranged from improving at- 
risk screening and referral strategies, to mentoring, to substance abuse prevention, to improving school 
security. 



Safe Schools Strategy 17 called for expanding the use of the NovaNET online learning system 
already available in some Wake County schools, to all 17 high school computer labs as well as 2 
alternative middle schools. Counselors and teachers recommended NovaNET to students struggling in 
traditional classrooms, in the belief that the alternative approach would increase student success, 
retention, and graduation; seniors who needed course credit in order to graduate received highest priority 
when labs were full. NovaNET offers a comprehensive range of high school curricula encompassing: 



English: 

Mathematics: 

Social Studies: 
Science: 

Foreign Language: 
Electives: 

SAT/ACT prep.: 



Grades 9 through 12 writing, literature, and reading 

fundamentals of mathematics, algebra I & n, geometry, trigonometry, 

advanced placement calculus, consumer mathematics 

U.S. government, U.S. history, world history, geography, economics 

anatomy & physiology, biology, earth science, environmental science, 

chemistry, advanced placement chemistry and physics 

French, Spanish, Latin, German, Japanese, Russian 

accounting, health & first aid, physical education, music, keyboarding, 

psychology, photography, critical thinking & study skills, life coping skills 

practice tests and curricula. 



Each of these curricular units is aligned with North Carolina’s Standard Course of Study. Although Wake 
County students do not participate in the more advanced or interactive aspects of the program (with the 
exception of a small number of educational games), NovaNET also includes national discussion and 
writing forums, links to numerous educational Internet resources, as well as a collection of fully 
searchable, commonly used reference documents. 



Each NovaNET lesson requires students to read instructional material provided on the computer 
screen followed by diagnostic questions (online); a few include written assignments. Students’ online 
answers are followed by new information, or further explanation, depending on the student response. 
Many students work without text support; neither NovaNET textbooks nor printed copies of lessons are 
available (either for use in class or to take home). When extra related texts are available at the school, lab 
coordinators secure copies for use in the computer lab. NovaNET courses used by Wake County students 
include no face-to-face interaction with an online instructor; when lab coordinators are available and 
familiar with the subject material, they may provide individual instructional support requested by 
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students. The program provides unit tests which determine 75% of the student’s grade; state end-of- 
course exams (EOC’s) account for 25% of the grade. Successful completion of the entire sequence of 
readings, questions, written assignments, unit tests, and the EOC comprises successful completion of the 
course credit, which may be applied to graduation requirements. Administrators at each school make the 
final determination of whether course credit may be assigned for NovaNET work. 

The Safe Schools goals for NovaNET address expanding the availability, binding, staffing, 
training, and awareness of the program, in order to increase its frequency of use. The first year of the 
program included in-service training for faculty, who were then responsible for identifying at-risk 
students and referring them to use NovaNET. At four county high schools, the program provided funds 
for computer lab coordinators responsible for working with students and tracking their progress with 
classroom teachers and students’ parents. At the other thirteen high schools and two alternative middle 
schools, lab coordinator positions were funded through alternative sources such as funds from the 
guidance or math departments, or were filled by volunteers; at six sites positions were shared or part-time. 
Currently in the second year of the three-year program, NovaNET is available in the computer labs of all 
Wake County high schools. Although the grant supplied 10 portals per school, schools have added 
portals when possible, bringing the Wake County total to over 250. 

School District Context 

Located in central North Carolina, Wake County consists of a blend of urban, suburban, and rural 
areas. Three universities, the state capital, 1 1 municipalities, one of the country’s largest research centers 
for business and industry, and numerous small farms are all located within its boundaries. The county is 
large (864 square miles), the current population of 600,000 placing it in the top 5% of counties 
nationwide. Wake County Public School System (WCPSS) is the second largest in the state, and the 30 th 
largest in the nation (97,600 students/122 school sites, 2001 WCPSS data). The area’s rapid population 
growth is reflected in Wake County’s standing as the third fastest growing district among the 40 largest 
school districts in the country. This growth, which is projected to continue, presents a staggering 
challenge to the school system’s commitment to meet the needs of all students. 

Not only is the size and growth of Wake County marked by extremes, the racial, cultural, and 
socio-economic diversity of the citizens and students is similarly extreme. Student population 
demographics (2001 WCPSS data) include 64% Caucasian, 27% African-American, 4% Hispanic, 3% 
Asian, and 2% Other. Contained within this diverse profile are over 100 different languages and cultures 
from all over the world. There are also pronounced socio-economic differences between the citizens of 
the county. Within Wake County, 1 1 .3% of school-aged children (16,282 children) live in poverty (1997 
U.S. Census), and 20.0% (19,516 students) are eligible for free/reduced lunch (2001 WCPSS data). 

Throughout the county, neighborhoods with high concentrations of at-risk children experience 
serious educational challenges. Although on average Wake County student achievement is exemplary 
compared to state averages, 15% of the student population (-15,000 students) performs below grade 
level. In 1999 there were 13,899 out-of-school suspensions or expulsions, 1,575 students on probation, 
and 2,756 students involved in the juvenile justice system (1999, North Carolina State Bureau of 
Investigation). In 1996 the countywide juvenile violent crime rate was 10.32 arrests per 1,000 children 
compared to 1 .82 for the state and 1 .34 for the nation. Due in part to the rapid population growth, 
extreme diversity, and other high risk factors, community wide resources and services struggle to meet 
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demands. Countywide, there are literally thousands of children needing resources and intervention; this 
number is increasing as the population grows. The merit of a program like Safe Schools, designed to 
coordinate efforts to serve the county’s student population using strategies such as NovaNET, is 
noteworthy. 

Evaluation Design 



The purpose of this evaluation is to describe NovaNET use and its effects during the second year 
of the Safe Schools program (2000-01). Table 1 presents the Evaluation Questions and Data Collection 
Strategies in an Evaluation Crosswalk (O’Sullivan, 1991). For each of the evaluation questions posed, 
multiple data sources were used to gather information. Two focus groups were conducted, one with 13 of 
the county’s 19 NovaNET computer lab coordinators, one with 13 of the 19 lead guidance counselors 
(referred to as student deans) who refer students to NovaNET. All lab coordinators and lead counselors 
(deans) were invited to the focus group meetings, which were scheduled by the county Research and 
Evaluation department; all attendees were included in data collection. The Research and Evaluation 
department also selected 10 lab coordinators (purposeful/non-random selection representing both well 
established and newer NovaNET programs) to participate in individual interviews (five of the interviewed 
lab coordinators had also participated in focus groups). Two guidance counselors and 25 students 
representing 4 schools, each of whom were available during school site visits, also participated in 
individual interviews (see Appendices A & B for protocols). Questionnaires were completed by 9 of the 
13 lab coordinators, and 9 of the 13 guidance counselors who attended focus groups (Appendices C & D). 
Four lab coordinators chose to conduct open-ended surveys concerning the impact of the program with all 
participating students at their school; one school surveyed all referring teachers at their school as well. 
Surveys were completed by 79 of 313 students at the four schools (some of whom also participated in 
interviews) and 1 8 referring teachers from one school (Appendices E, F & G). 



Table l. NovaNet evaluation Crosswalk 



Issues 


Focus 

Groups 


Inter- 

views 


Question- 

naires 


Surveys 


System 

Data 


1 . Were projections set and met regarding number of 
students participating in NovaNET? 










X 


2. Does the average ABLE grade level score increase 
following student participation in NovaNET? 










X 


3. What is the average amount of time spent per student 
using NovaNET? Is there a general trend among schools? 










X 


4. In what ways do student participants of NovaNET feel they 
have or have not benefited from NovaNET? 




X 


X 


X 




5. In what ways do NovaNet coordinators believe NovaNet 
benefits or does not benefit student participants? 


X 


X 


X 






6. How are students selected to participate in NovaNET? Is 
there a general trend from school to school? 


X 


X 


X 


X 




7. How is NovaNET implemented at each campus? Is there a 
general trend among schools? 


X 


X 


X 






8. What problems have NovaNET coordinators experienced? 
How severe are the problems? Are they easily solved? 


X 


X 


X 






9. Are overall gender, ethnicity, language, and special 
program demographics of student participants 
representative of the overall WCPSS student population? 










X 
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Results 



1. Were projections set and met regarding numbers of students participating in 
NovaNET? 



Initial goals for the Safe Schools initiative projected that 600 high school students across the 
county would take part in NovaNET on-line learning during the second year of the grant. This goal was 
exceeded substantially as more than 1300 students attempted at least one NovaNET course at 15 high 
schools, 2 alternative high schools, and 2 alternative middle schools (see Table 2). A second NovaNET 
course was attempted by 485 of the participants (35.4 %); 24 attempted a third course (1.7 %). A total of 
1878 courses were attempted by the pool of students attempting 1, 2, or 3 courses. 

Typical catalysts triggering referral to NovaNET were failure in traditional classes, serious 
attendance issues, and personal issues such as illness or pregnancy. Seniors requiring course credit in 
order to graduate were consistently given priority when labs were full and access to portals was 
limited. Students who transferred to Wake County during high school were also offered NovaNET as 
a means to make up coursework needed to meet state graduation requirements. 

Designers of the Safe Schools program had also anticipated that NovaNET would prove useful to 
the nearly 3,000 Wake County students involved in the Juvenile Justice system. Only three Juvenile 
Justice students were included in the official, year-end, NovaNET data required by the central office and 
submitted by lab coordinators. The lack of participation by this population is being addressed currently 
by the school system, with attention being given to careful coordination between Juvenile Justice 
personnel and NovaNET lab coordinators. Such attention should better serve the Safe Schools agenda by 
involving more of the students for whom Strategy 17 was specifically designed. 

WCPSS additionally projected that 80% of attempted credits would be successfully completed. 
As seen in Table 3 A, an average of 73.8% of students system-wide were successfully completed, 
although the results varied widely across subject area. Students undertaking Elective courses (94.4% 
attempted credits earned), Social Studies (80.6% credits earned) and Math courses (79.7% credits earned) 
were successful at the projected levels. English students were slightly less successful (68.4% of 
attempted credits earned), and only 48.6% of Science students successfully completed course work. 
Among the 1,899 NovaNET courses attempted, 839 (50.3%) enabled students to recover “on time for 
graduation” status, and 417 (25.0%) moved students closer to graduation (having earned credits needed 
for graduation but not enough to be on track to graduate with their class). An additional 41 1 (24.7%) have 
not made progress toward graduation, due either to receiving an Incomplete (n=28) or a failing grade 
(n=383). (Data was missing for 232 courses.) Table 3 B illustrates the impact on graduation status of 
each NovaNET course attempted. (Wake County schools require between 20-24 credits for graduation, 
depending on the school and program.) 
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Table 2: NovaNET Status by School 
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Table 3: Progress Toward Graduation 



3. A. Credits Earned Toward Graduation 



Subject 


Credits Attempted 


Credits Earned 


% Credits Earned 


Elective 


108 


102 


94.4 % 


Social Studies 


232 


187 


80.6 % 


Math 


605 


482 


79.7 % 


English 


326 


223 


68.4 % 


Science 


220 


107 


48.6 % 


System 


1491 


1101 


73.8 % 



Incompletes (n=28) not included in total. 



3. B. Student Graduation Progress per NovaNET Course 

(all courses) 



Eligibility 


Frequency 


% 


On track for graduation 


839 


50.3 % 


Closer to graduation 


417 


25.0 % 


Data missing 


411 


24.7 % 


Total 


1667 


100.0% 




As illustrated in Figure 1, the grades for courses completed successfully were widely distributed. 
Students were most successful earning passing grades in electives (94%) and least successful in science 
(48%). 
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Figure 1 : Profile of Student Success in NovaNET Credit Courses 

(Course Grade) 
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2. Does the average Adult Basic Learning Examination (ABLE) grade level score 

increase following student participation in NovaNET? /What is the average grade 
level score change in ABLE Scores following student participation in NovaNET? 



The Safe Schools program delineated specific pre- and post-intervention evaluation measures. 
Strategy 17 included the Adult Basic Learning Examinations (ABLE) administered to NovaNET students 
at the outset and near completion of their NovaNET coursework, although many students missed either 
the pre- or post-test. ABLE is a standardized, multiple-choice, achievement battery with sub-tests in 
Vocabulary, Reading Comprehension, Spelling, Language, Number Operations, and Problem Solving. 
ABLE results provide grade level scores (year and month) of students’ performance on each sub-test. As 
shown in Figure 2, changes in ABLE scores varied by subcategory, with Vocabulary and Numerical 
Operations improving significantly in the expected direction (Vocab. t = 4.154, df 169, p<.001; 

Numerical Operations t = 4.263, df 174, p <.00 1). Problem Solving, however, declined significantly (t = - 
2.004, df 178, p < .05). The sample size and variance of scores varied widely among subtests. This 
discrepancy should be kept in mind when interpreting the significance of differences as classical tests for 
the statistical significance of differences between scores are affected by both variations in sample size and 
differences in variability of scores. It is also important to note that tests could only be completed for 
students for whom both pre- and post-test measures were included as part of the year end data report filed 
by lab coordinators to the central office; the sample size for this comparison is substantially smaller than 
the entire group of NovaNET participants. 



Figure 2: Pre- vs. Post- Mean ABLE Sub-test Score 
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3. What is the average amount of time spent per student using NovaNET? Is there a 
general trend among schools? 



As seen in Table 4, hours invested in each NovaNET subject varied widely across the 19 
schools, with average time online ranging from 17.03 - 45.01 hours. Across all schools, the highest 
average time invested was in Science ( X = 45.0 1 hours), Social Studies ( X = 43.8 1 hours), and Math 
( X= 40.80 hours), contrasted with the least amount of time expended on English (X= 30.43 hours) 
and Electives ( X = 17.03 hours). Table 5 demonstrates the wide variety in the pattern of time spent 
on each subject at each individual school. No significant pattern among schools emerged. 



Table 4: Hours On Line by Subject 



Subject 


n 


Mean hours 
spent on line 


Minimum 


Maximum 


Science 


220 


45.01 


1.00 


209.83 


Social Studies 


232 


43.81 


0.65 


327.00 


Math 


605 


40.80 


0.50 


324.33 


English 


326 


30.43 


1.25 


162.67 


Electives 


108 


17.03 


0.25 


154.00 













Table 5: Average Hours On Line by Subject by School 



School Name 


Average 

Time 

Online 


Science 

Time 

Online 


Soc.Stud. 

Time 

Online 


Math 

Time 

Online 


English 

Time 

Online 


Elective 

Time 

Online 


1. Mary E. Phillips High 


77.54 


75.53 


79.77 


76.58 


58.82 


97.00 


2. Apex High 


76.87 


86.69 


75.97 


115.22 


64.17 


42.32 


3. Fuquay-Varina High 


53.75 


46.50 


37.00 


58.5 




73.00 


4. Wakefield High 


51.00 


- 


- 


68.00 


34.00 




5. Green Hope High 


47.73 


55.89 


40.90 


61.65 


40.65 


39.57 


6. Sanderson 


44.60 


. 30.00 


96.00 


50.57 


24.62 


21.80 


7. Cary High 


42.10 


56.67 


37.18 


43.10 


. 


31.45 


8. Millbrook High 


39.39 


41.00 


45.00 


39.83 


39.36 


31.75 


9. Southeast Raleigh High 


38.38 


41.31 


42.66 


49.46 


32.10 


26.39 


10. Garner High 


37.83 


46.37 


43.81 


46.31 


33.57 


19.09 • 


1 1 . Athens Drive High 


31.45 


31.12 


33.03 


35.33 


26.30 




12. East Wake High 


25.70 


65.50 


- 


13.56 


18.08 


5.65 


13. Broughton High 


24.71 


23.69 


24.46 


29.22 


30.38 


15.81 


14. Leesville Road High 


22.67 


59.89 


17.61 


6.25 


6.93 




15. Richard Milburn High 


20.50 


22.86 


19.33 


24.52 


15.80 


20.00 


16. Wake Forest-Rolesville High 


16.41 


15.75 


3.50 


22.14 


24.24 




17. Longview Middle/High (6-12) 


15.83 


6.75 


. 


43.30 


19.25 




18. Enloe High 


9.61 


5.75 


16.21 


9.88 


8.90 


7 31 


19. Mt. Vernon Redirection (6-8) 


Data missing 


Data missing 


Data missing 


Data missing 


Data missing 


Data missing 

















no course recorded 



As seen in Table 6, exploring the relationship between time on line and either post-course 
grades or course credit earned highlights a modest result of the NovaNET program. A small, but 
statistically significant, positive correlation exists between time on line and grades for Science, Social 
Studies, Math, and English; no such relationship exists for Electives. Progress made toward 
graduation (credits earned) has a similarly modest positive correlation with time spent on line. 



Table 6: Hours On Line by Post-Course Grade 



Subject 


n 


Correlation (R) 

Time Online x Course Grade 


Sig. 


Science 


220 


.26 


p< .01 


Social Studies 


232 


.18 


p< .05 


Math 


605 


.14 


p< .01 


English 


326 


.19 


p< .05 


Electives 


108 


.004 


not sig. 






Time Online x Credit Earned 




Across subjects 


1491 


.15 


p< .001 











4. In what ways do student participants of NovaNET feel they have or have not 
benefited from NovaNET? 



“NovaNET is the best thing that happened to me during my high school career. 

I believe that it is the only reason I want to go to college... 

I learned how to study and learn , that in order to succeed you have to ask questions, 
that I really do need to do my homework, 
and that “ checks and balances " don 7 fit in my purse... 

I will never forget everything NovaNET has done for my future, 
and I thank you truly for everything you have done and gone through just for me. ” 

Student participant 

NovaNET appeals to students for a variety of reasons. Statistics, test scores, and numbers of 
participants only begin to capture the value of NovaNET in the lives of students. Sentiments like those of 
the NovaNET participant quoted above reveal many dimensions of the impact of the program. Not only 
did students learn, they often learned how to learn, and occasionally they were reminded that they were 
capable of learning. The student quoted above was a junior who was failing prior to her experience with 
NovaNET; currently in her senior year, she is on the AB Honor Roll. Like this individual, the program 
participants consulted in 25 interviews and 79 surveys consistently referred to relational, academic, or 
personal benefits that reach far beyond the subject knowledge they acquired. Data from 18 teachers 
(surveys), 18 lab coordinators (13 focus group/10 interviews/5 both), and 15 guidance counselors (13 
focus group/2 surveys) richly echoed these stories of impact on student lives. 

As illustrated in Table 7, most commonly, students identified the ability to “work at their own 
pace” as what they liked best about NovaNET. Out of the 104 students who participated in interviews 
(25) or returned surveys (79), 83 (80%) highlighted this benefit. As a result of the relaxed pace, they 
claimed to feel less stressed, less rushed, and less frustrated in the lab than in their traditional classes. A 
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small number of students (5%) emphasized that this made the lab a fun, healthy, relaxing learning 
environment. 



Table 7: NovaNET Benefits - The Students' Perspective 

Benefits of the Learning Environment 
Self-pacing (80%) 

Teacher Encouragement (38%) 

Examples: NovaNET teacher became an advocate for students. 

Students believe their NovaNET teacher cares. 

The students receive individualized attention. 

A second chance, the ability to catch up (37%) 

A fun, healthy, relaxing learning environment (5%) 

Program Effects 

Graduation (33%) 

Prevention of drop-outs 
Recovery of on-time graduation 
Motivation & Self esteem (5%) 

Consider college for the first time 

Students feel hopeful, see a way out, see light at end of tunnel 
Builds self confidence 
Improves self-image 



Second in response frequency (38%) was the importance of the consistent, persistent, 
individualized support students received from their NovaNET teacher. The focused attention they 
received allowed them to build a friendly, supportive relationship. One student commented that, 
compared to her NovaNET instructor, teachers in her other classes " don 7 even know me .” One lab 
coordinator suggested that, in NovaNET, students found a teacher who said, “lam not going to let you 
fail . ” Thriving in this setting, students often succeeded academically, gained self-confidence, and grew in 
their understanding of and engagement in the process of learning. One lab coordinator highlighted the 
importance of this individualized attention when she cautioned. against allowing the program to grow too 
rapidly, lest the program’s ability to attend individually to students be compromised. 

NovaNET participants also commented frequently (37%) that NovaNET provided them with a 
second chance, the ability to learn from their mistakes. Unlike many traditional classrooms, in the 
computer lab students were able to re-take assignments and tests until they understood and mastered the 
material. They also enjoyed the lack of homework and the open book tests. 

The NovaNET learning experience had direct effects on students’ lives and attitudes. Many 
(33%) articulated that NovaNET allowed them to catch up and graduate with their class. One student 
described the lesson she learned this way, “It is a great second chance for any student who may have 
messed around or slacked off in a class or two. And it helped me realize that you don 7 always get a 
second chance, so you should take care of your business the first time. ” A small number of students (5%) 
also were aware that NovaNET influenced their motivation and attitude about life, school, and 
themselves. “ NovaNET empowers you to attain the goals you have set for yourself” said one student. 
Others claimed, “Some people think that they can 7 do well in school, so they give up on themselves. This 
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program makes kids get the help they need, ” and “The only way to get a bad grade in NovaNET is if you 
just don 7 care. ” One described the impact of his NovaNET experience this way: 



For the first time in high school I enjoyed taking math. NovaNET made me more 
motivated. I wish I had been able to be a part of the program previous to this year. I 
enjoyed working at my own pace and having someone available to help me when I need 
it. Taking NovaNET made me realize I could get better than a C in Math and realizing 
that motivated me to work a little harder and get an A. , 

Seven percent of program participants offered suggestions for improving the NovaNET 
program. Students agreed that their interest and motivation to use NovaNET would have been 
enhanced by a more interactive design and more graphics, rather than the version they had used, 
which offered little or no graphics, color, realism, or interaction. Four students recommended a 
Windows version of the program with “point and click” options, rather than the syntax version they 
used which required knowledge of the precise verbal commands accepted by the program. Six 
students commented that having access to written materials coordinated with the NovaNET lessons or 
the ability to print lessons would be helpful. Students also were frustrated by the “pickiness” of the 
program, which did not accept reasonable synonyms for lesson vocabulary, or alternative 
mathematical notation. 

One important potential application of NovaNET is with students who are homebound. Such 
students may be capable of completing schoolwork, but not able to attend traditional classes. One of 
1 3 homebound students participating in NovaNET was interviewed extensively. She described 
NovaNET as a “lifeline” but also suggested that students like herself, dependent on NovaNET for 
major portions of their educational experience, needed state-of-the-art computer equipment to 
minimize the delays and frustrations caused by problems with technology. She also reflected that 
access to text materials and the ability to print lessons are especially important to students outside of 
the support structure of the school setting. The ability to stop and rest without the program timing out 
and shutting down would be helpful to students dealing with illnesses. As with other adolescents, 
relationships are of key importance to homebound high school students; a chat room or discussion 
format (student-to-student as well as student-to-teacher) would be very valuable. 



5. In what ways do NovaNET coordinators, teachers and guidance counselors believe 
NovaNET benefits or does not benefit student participants? 



As reflected in Table 8, the 18 lab coordinators, 18 teachers, and 15 guidance counselors 
consulted believed that NovaNET most benefited students by preventing drop outs, providing a second 
chance for students to catch-up or improve failing grades, and improving student outlooks on their 
education. By far the most frequently mentioned benefit of NovaNET by adult respondents in this study 
(38 of the 5 1 respondents) was the prevention of drop outs, through providing students with a second 
chance to recover their high school diploma. Cases of successful graduation recovery credited entirely to 
NovaNET were highlighted by 19 out of 51 professionals participating in the study. Several lab 
coordinators told stories of NovaNET graduates who were the first in their family to receive a high school 
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diploma. The program was described by one guidance counselor as their high school’s "most powerful 
drop-out prevention tool! ” 

Thi rteen teachers and lab coordinators specified the second chance that NovaNET offered 
students as an important reason for its positive impact on students. One social worker reflecting on this 
opportunity to recover or catch-up summarized her thoughts this way: “7/ has allowed many students to 
pull themselves out of a pretty deep hole and go on from there.” 



Beyond practical issues such as graduation, one word resonated throughout the comments of adult 
participants in this study regarding the impact of NovaNET on students. That word was HOPE. School 
personnel (13) spontaneously used the word hope to refer to the rejuvenating effect of NovaNET on 
student outlooks. One lab coordinator explained, 

The students feel more hopeful when they can feel better about one part of their life. 

It has a ripple effect. They no longer feel like something is hanging over their head, 
and can see the light at the end of the tunnel. Then they can think more clearly, set 
goals, show discipline...” 

Lab coordinators and counselors described other dimensions of the impact of NovaNET that went . 
beyond the knowledge students learned in computer sessions, such as increasing student self-confidence 
and study skills. A teacher referring students to NovaNET stated, “NovaNET students have accomplished 
more than they thought they could and more than others believed them capable of.” Another individual 
reflected, “It is amazing what it does for their self confidence.” One lab coordinator echoed these 
thoughts with, "7 have seen students do things with NovaNET that they never did in class. It is amazing!” 
These quotes, all from individuals who have directly observed the benefits participants have received, 
sketch a profile of the potential impact of this innovative program. 

All of the lab coordinators interviewed (10) concurred that students who remained unmotivated 
had continued attendance problems, were unable to focus and work independently, and lacked reading 
skills would not benefit or would struggle with NovaNET. While they insisted that the program was not 
for everyone, they acknowledged it as an important asset to many students not thriving in traditional 
classrooms. In the arsenal of interventions for at-risk students, the lab coordinators interviewed awarded 
NovaNET their highest rating. 



Table 8: NovaNET Benefits — 

The Lab Coordinators’ Perspective 

Prevention of Drop Outs (Highlighted by 38 of 46 individuals interviewed) 
Provision of a second chance for students (13) 

Improvement in student encouragement / hope (13) 

Students who remain unmotivated, continue attendance problems, are 
unable to focus and work independently, and students with unusually poor 
reading skills will not benefit from NovaNET. 
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6. How are students selected to participate in NovaNET? Is there a general trend from 
school to school? 



As seen in Table 2, among schools with well developed, long standing, and well staffed 
NovaNET programs such as Cary High School, Athens Drive High School, Gamer High School, and 
Apex High School, the process of referring students to the NovaNET lab was also well-developed. In 
such schools, guidance counselors, teachers, and social workers were informed about NovaNET and 
referred students on a regular basis. At these schools, the students were also aware of the program 
and asked to participate. These well-developed referral mechanisms, as well as the high profile of 
NovaNET at such schools, typically resulted in higher rates of participation. 

Schools varied in the degree to which systems were in place to adequately identify all 
students who might benefit from NovaNET. Several schools were late in initiating their program 
(Sanderson, Longview), did not have full time NovaNET staff (Broughton), or lacked administrative 
support (East Wake High). Personnel at these schools indicated that it would only be during the third 
year of the program that they would fully implement a well developed referral process, or teacher 
orientation strategy, that would allow the program to meaningfully funnel the maximum number of 
appropriate candidates into the NovaNET lab. The time required to develop a system that worked 
well for their individual schools, coupled with the lack of personnel available to invest time in the 
program, and lack of administrative support, were the reasons given by six lab coordinators for 
making slow progress with NovaNET program setup. 

At all schools countywide, the primary referral mechanism for NovaNET was the school 
guidance program. The most frequent referrals to NovaNET, according to students as well as lab 
coordinators (over 80%) came from counselor recommendations. Typical catalysts alerting 
counselors to consider NovaNET referral were failure in traditional classes, serious attendance issues, 
and personal issues such as illness or pregnancy. Seniors who needed course credit in order to 
graduate received highest priority when labs were full. Students who transferred to Wake County 
during high school were also offered NovaNET as a means to make up coursework needed to meet 
state graduation requirements. 

As teachers learned more about the NovaNET opportunity and developed confidence in its 
results, referrals began to originate within the classroom. The degree of teacher awareness and 
confidence in NovaNET varied across schools. Typically, strong administrative support and the 
availability of a full time lab coordinator heighten the program’s profile. At Cary High School, for 
example, with a well-established and well -supported NovaNET program, teachers regularly refer 
students and comment about the value of the program. The success of the program, both in terms of 
numbers of students participating and numbers of students graduating as a result of NovaNET is 
consequently high. 
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7. How is NovaNET implemented at each campus? Is there a general trend among 
schools? 



Where NovaNET had been actively developed, it was typically used widely and creatively 
(See Table 2). These programs often had standing room only and waiting lists. Their chief concern 
about NovaNET was the need for more portals. Where administrative support or staffing was 
missing, or where schools had experienced moves, administrative changes, or other barriers to 
implementation, the success of the program was limited. Seven of the county’s high schools made 
limited use of NovaNET - primarily for enrichment, course competency, and test preparation - not as 
a serious tool for course recovery or remediation. Four labs were not yet established due to space or 
staffing issues, or were just getting started at the time of this study 

Eight reporting schools noted using NovaNET (or beginning to employ it) during every 
period of the day. The busiest made it available during lunch as well as during either alpha or omega 
periods (before or after school) and in the evening. Four schools reported integrating NovaNET into 
summer school programs. Many would like to see it used more extensively, expanded into middle 
schools, as well as concurrently with traditional classroom instruction, as a bulwark to prevent 
students from experiencing failure in standard classrooms. 

The eight well established programs reported that the predominant NovaNET uses were 
recovery of course credit toward graduation or repairing a failing grade from a traditional course 
(remediation). Seniors who needed course credit in order to graduate received highest priority when 
labs were full. NovaNET was also frequently used to prepare for end-of-course exams and 
standardized tests, for electives, and for enrichment. Three schools reported using NovaNET 
successfully with students who were homebound for medical reasons; one school allowed special 
education students to use the program under their teacher’s supervision. 

The greatest variation in implementation resulted from the diverse and often resourceful 
attempts to staff the NovaNET lab. NovaNET personnel came from all walks, from varied teaching 
backgrounds, out of Guidance or Math departments, or through the ranks of volunteers and student 
interns. Schools that did not receive Safe Schools funds to underwrite a full time lab coordinator have 
allocated resources from a wide range of programs. Frequently two people share the position of 
administering the program (in combinations of lab coordinators and volunteers, lab coordinators and 
teachers, lab coordinators and guidance counselors, or 2 lab coordinators). NovaNET staffing 
arrangements therefore ranged from a full time coordinator supported by three academic specialists, 
to part-time teachers working with volunteers. This diversity drastically affected the school 
community’s awareness of the program, the number of students referred, the schedule of NovaNET 
availability, and teacher support for NovaNET students. 
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8. What problems have NovaNET coordinators experienced? How severe are the 
problems? Are they easily solved? 



As reflected in Table 9, a variety of problems were expressed by NovaNET lab coordinators, 
ranging from technical support requests, to staffing needs, to policy issues. None of the struggles 
reported were severe or insurmountable. Supporting the necessary staff to develop this intervention 
program was the greatest challenge. Beyond this financial demand, solving the problems and supplying 
the needs articulated by NovaNET staff appear to be attainable goals. One problem that those 
implementing NovaNET were NOT experiencing was with service from NovaNET developers. At every 
school, lab personnel noted prompt and efficient customer support. 

Staffing issues were clearly the most salient struggle that lab coordinators experienced. Personnel 
from every school (100%) communicated the absolute necessity of having a full-time staff member to 
administer NovaNET - someone with the responsibility to oversee and coordinate the program, manage 
the assignment of course credit, and “sell” NovaNET to the faculty. Given the high school subject matter 
of NovaNET courses, all lab coordinators experienced the necessity of access to subject area specialists. 
Fourteen of 18 coordinators (13 in focus group, 10 interviewed, 5 both) suggested that truly successful 
implementation required the availability of at least two staff positions: a coordinator to manage the 
program and a teacher to work with the students in the lab. Both tasks are demanding and imperative to 
the success of the program. NovaNET coordinator positions were funded at 4 high schools through the 
Safe Schools program. At other schools, positions have been funded through alternative sources such as 
the Guidance or Math Departments; others were staffed with volunteers. Where no full time NovaNET 
position exists, personnel seemed at a loss as to how to add NovaNET to their already extensive 
responsibilities, and the program struggled. Another serious detriment to programs without NovaNET 
staff was that they had no “cheerleader” for the program - no one supplying energy, enthusiasm, and 
drive for its success. 

Even two years into the Safe Schools program, many lab coordinators still needed help getting 
started, were experiencing problems developing referral processes, system administration, and/or tracking 
course credit. Half of the county’s high schools did not actually begin their NovaNET program until the 
2000-200 1 school year, some during the second semester of that year. The demands of their other 
responsibilities as well as the immensity of initiating a NovaNET program made the task daunting. The 
need for uniform NovaNET policy and credit guidelines, facilitated by a lead, county-wide, NovaNET 
coordinator to assist and coordinate implementation, was suggested by three coordinators. Currently those 
decisions are made at the school level, the responsibility of either the administrator or lab coordinator. 
Although assistance in each of these areas would facilitate program development (as suggested by 8 lab 
coordinators), late starters were confident that the third year of the program would yield great results for 
the students at their schools. 

Another concern expressed by seven of the 1 8 lab coordinators surveyed and interviewed was the 
need for more NovaNET portals. At the eight sites where the program was used extensively, labs were 
bursting at the seams, used every period of the day, and still students were waiting for terminals. 

Computer lab personnel in these labs were forced to turn students away. 
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The need for administrative and teacher support was also a concern for six lab coordinators. One 
lab coordinator speculated that one could predict the success of the NovaNET program at any given 
school based on administrative/faculty support alone. 

Lab coordinators less frequently mentioned technical or program concerns such as wiring 
problems, work disappearing on line, the inability for students to stop and rest without the program timing 
out and shutting down, and the desire for students to check their own grades. Four lab coordinators 
mentioned problems with the design of the program, such as the exactness of program vocabulary (not 
accepting reasonable synonyms) and the lack of interaction, animation or graphics. Two felt the need for 
further training in using NovaNET reports and other technicalities of the program. Also, without text or 
face-to-face support, the only way for students to receive additional information is to solve a problem 
incorrectly which prompts the program to provide additional instruction. 



At least one full time NovaNET lab coordinator at each school (18) 

Subject area specialists available to computer lab (18) 

Help getting started. / Help designing efficient, effective referral process (8) 
Uniform policy / credit guidelines system-wide. 

More NovaNET portals in labs with waiting lists and/or standing room only (7) 
Administrative support for funding, staffing, (6) 

approval to administer course credit for work completed on line, etc. 
Programming suggestions (4) 

Students should be able to view own grades 
EOC’s should be available on line 
Ability to stop and take a break without program timing out 
More interactive and attractive platform 
Technical assistance: (2) 

Further training 
Help with reports 

Problems with work disappearing in the system 



9. Are overall gender, ethnicity, language, and special programs demographics of 
student participants representative of the overall WCPSS student population? 



With over 100 language and cultural groups, and pronounced socio-economic differences, Wake 
County is a site of remarkable diversity. Administering a program across such a varied population holds 
many challenges. Exploring the demographic characteristics of NovaNET participants compared to the 
countywide system does reveal similarities as well as contrasts. Although the proportion of language 
minority students employing NovaNET (1.0% compared to 4.0% system-wide) mirrors the proportion in 
the larger system, comparisons of gender, ethnicity, and educational classifications, as well as the 
proportion of participating students who qualify for free/reduced lunch, reveal differences. 



Table 9: Problems/needs experienced with NovaNET 

(18 Lab Coordinators interviewed and/or surveyed) 
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As illustrated in Table 10 A, about 10% more males participate in NovaNET (60.3%) than their 
system-wide proportions (50.9%); female representation (39.7%) is 10% lower than system wide (49.1%) 
(A 2 = 44.72, df = 1, P < 001). African Americans also participate in greater proportions than their system- 
wide percentages (47.1% vs. 26.2% respectively, A 2 = 346.40, df = 4, P < 001). Given that NovaNET 
staff have reported that the program works well for at-risk students, these proportions could parallel the 
over-representation of African American males in the Wake County at-risk population. The proportion of 
NovaNET students who qualify for free or reduced lunch is also high compared to the proportion among 
high school students system wide (91.4% vs 20.0% system-wide, A 2 = 895.40, df = 1, p<.001). With free 
lunch data missing for 80% of cases, the accuracy of this statistic is uncertain. Table 10 B illustrates the 
proportions of NovaNET students with various educational classifications compared to the system at 
large. Although many special needs classifications represented in NovaNET students are parallel with the 
larger system, the proportion of NovaNET Learning Disabled students is higher than the system at large. 



Table 10 : Demographics 
N ovaNET Participants vs. System-wide 



10. A. Gender & Ethnicity & Free Lunch Status 



Gender 


# 


%of 

NovaNET 

participants 


Percent 
system wide 

(’99 WCPSS data) 


Male 


776 


59.5 % 


50.9 % 


Female 


511 


39.2 % 


49.1 % 


Ethnicity 








African American 


614 


47.1 % 


26.2 % 


European American 


574 


44.0 % 


63.2 % 


Hispanic 


63 


4.8 % 


4.6 % 


Asian 


18 


1.4% 


3.9 % 


Other / Mixed 


35 


2.7 % 


2.1 % 


Free/Reduced Lunch Status 








Free or Reduced Lunch 


256 


91.4% 


20.0 % 


Total 


1304 


100.0% 


100.0% 



Gender data missing: 82 cases; Ethnicity ;data missing: 65 cases; Free Lunch data missing: 1,113 cases. 



10. B. Educational Classification 



Educational Classification 


# 


% of 

NovaNET 

participants 


Percent 
system wide 

(’99 WCPSS data) 


Non-classified 


964 


69.9% 


71.4% 


Learning Disabled 


212 


16.4% 


6.6 % 


Behavioral Emotional Disabled 


33 


2.5 % 


0.7 % 


Educable Mentally Disabled 


12 


0.9 % 


0.8 % 


V 

Speech/Language Impaired 


6 


0.5 % 


2.0 % 


Other Classification 


77 


5.7 % 


3.2 % 


Total 


1304 


100.0% 


100.0% 



Educational classification data missing for 65 cases. 
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Summary 



The current evaluation provides a variety of insights into NovaNET online learning as a part of 
the Wake County Safe Schools’ effort to provide optimal learning environments for all students. 
Although the newness of the program limits the size of its influence to date, where staffing, support, time, 
and space have allowed, NovaNET is used creatively, extensively, and with encouraging impact, as the 
following summary of positive results suggests. 

> More than twice the projected number of students participated in NovaNET; 

> 73.8% of attempted credits earned were successfully completed; 

> 843 students are back on track for graduation as a result of NovaNET after being off- 
track for graduation with their class prior to NovaNET; 

y Teachers consider NovaNET to be a great drop-out prevention tool; 

> Teachers report that NovaNET improved student motivation, self-esteem, and 
hopefulness; 

> Student-perceived advantages of NovaNET included: 

■ self-pacing of the program, 

■ one-on-one teacher attention, 

■ a chance to catch up or recover academically, and 

■ relaxing learning environment. 

> NovaNET provided excellent customer support. 

As with any new and innovative program presented to busy, already committed teachers, 
NovaNET has taken time to get established in Wake County. Several schools were just beginning 
their NovaNET program during the semester this evaluation took place (two years into the Safe 
Schools grant). Several more are actively involved in the program, but were still refining their 
processes. Several factors were consistently believed to directly impact implementation. 

y All schools concurred that a full-time NovaNET coordinator is vital to the success of 
the program. 

> Lack of access to subject area teachers made learning challenging subjects difficult. 

> Many programs needed help getting started (i.e., help designing their referral process, 
understanding all aspects of the program, designing and managing a uniform course 
credit policy...) 



> Insufficient numbers of portals forced busy labs to turn students away. 

> Lack of administrator support was a challenge to any site. 

> Lack of textbook support coordinated with NovaNET lessons limited student 
learning. 

> Only three of the nearly 3,000 Wake County students involved in the Juvenile Justice 
system have participated in NovaNET. 

> Wake County students do not have access to a variety of innovative, resourceful, and 
interactive programs (discussion forums, online, face-to-face access to instructors...) 
available in the NovaNET program. 

Suggestions for improving the implementation of NovaNET, according to lab coordinators 
and students, include: 

> A full time lab coordinator and instructional specialist are needed for each site. 

> A Windows (rather than Syntax) version of NovaNET, along with, more graphics, and 
online interaction with an instructor would increase ease of use and student 
motivation. 

> Countywide collaboration concerning NovaNET procedures and policies (i.e., 
recruitment and assigning course credit) would allow newer programs to leam from 
those more established. 

> Adding NovaNET to evening, summer school, and middle school classes. 

Even with lack of full deployment, the NovaNET program is considered a vital intervention 
strategy by lab coordinators, teachers, guidance counselors, and students. Most NovaNET 
coordinators are excited about the third year of the Safe Schools initiative, when they believe they 
will finally be ready to employ NovaNET effectively. Evaluating the use and impact of NovaNET 
during the 2001-2002 school year is therefore vital to capturing an accurate portrait of the 
effectiveness of Safe Schools Strategy 17 and the value of online learning in reaching the educational 
goals of the Wake County Public School System. 



NovaNET is a way to offer hope... and hope is hard to come by 
for the student who has experienced repeated failures. 

Social Worker 

You call it NovaNET. I call it a second chance for students 
who otherwise would be casualties of our educational system. 

Computer lab coordinator 
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Appendix A 



NovaNET Interview - Lab Coordinators 

To what degree is NovaNET used at your school? 

How many computers are currently running NovaNET? 

During what hours/times of the day/week is NovaNET used? 

With what frequency as NovaNET used? (i.e. hours, number of students,. . .) 

In what ways is NovaNET used? (course recovery, remediation, test prep., EOC’s, ALP,. . .) 
Is NovaNET used for courses required for graduation? 

Is NovaNET used for content courses or otherwise? 

How are students referred to NovaNET? 

What criteria are used to select students for NovaNET? 

Who usually refers students to the lab for NovaNET? 

What impact has NovaNET had on students? 

In what specific ways have NovaNET students benefited from NovaNET? 

What kind of student does not benefit from NovaNET? 

Can you share an example of a student who has had a successful experience with NovaNET? 
How does NovaNET compare to other help available to students? 

What problems do you have with the NovaNET program? 

What one thing would help the most to increase NovaNET success at your school? 

What recommendations do you have regarding NovaNET? 
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NovaNET Interview - Students 



How were you referred to NovaNET? 



What do you like about using NovaNET? 



What do you NOT like about using NovaNET? 



How have you benefited from NovaNET? 



What characteristics of NovaNET help you be more successful as a student? 



Can you share an example of how NovaNET has helped you in school? 



What one thing would help the most to improve NovaNET? 
(to make you like it more or make it more helpful) 



What recommendations do you have regarding NovaNET? 
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Name 



Appendix C 



NovaNET Questionnaire (Computer Lab Coordinators) 



School Phone/email 



To what degree is NovaNET used at your school? (i.e., frequency, number of 
students,...) 

In what ways is NovaNET used? (i.e., core course recovery, electives, EOC 
competency,,...) 



What affects whether NovaNET is used? (i.e., teachers, time, computer skill,. . .) 
What impact has NovaNET had on students/teachers? 



How does NovaNET compare to other help available to students? 

What are the primary reasons that students struggle and/or fail? 

What one thing would help the most to increase NovaNET use at your school? 
What recommendations do you have regarding NovaNET? 



O 
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Appendix D 



NovaNET Questionnaire (Lead Counselors/Student Deans) 



Name 



School 



.Phone/email 



To what degree is NovaNET used at your school? 



What affects whether NovaNET is used? 



What impact has NovaNET had on students/teachers? 



What are the primary reasons that students struggle and/or fail? 



Appendix £ 

NovaNET Survey # 1 

(Designed by NovaNET Lab coordinator) 



Please take a moment to give me a comment on your evaluation of the impact of 
the NovaNET program at your school. Discuss what you believe to be the strong 
points of the program. 
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NovaNET Survey # 2 

(Designed by NovaNET Lab coordinator) 

NovaNET Feedback Form 



Student name 

Two ways that NovaNET is positive for me are: 

1 . 

2 . 

Two ways the NovaNET could be improved for me are: 

1 . 

2 . 

The midterm and/or final exam on NovaNET was: 

1 • easier than a traditional paper exam 

2. more difficult than a traditional paper exam. 

Comments about the exam or the program which you would like to share: 

Thank you for your input. 



O 
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Appendix G 



NovaNET Survey # 3 

(Designed by NovaNET Lab coordinator) 

NovaNET Lab Student Participation Questionnaire 

Please respond honestly to the following questions. 

You do not have to answer in complete sentences but you do have to answer each question. 



Name 

Age Birthdate Classification 

Course taking in NovaNET 

Counselor Academic Coach 



1 . Describe your experience in the regular classroom setting. 




2 . 



What are your academic goals after graduation? 



3. How were you referred to the NovaNET program? 



4. Is this a program that should be funded next year? Why or why not? 



5. Why are you successful or unsuccessful at school in general? 



6. In what ways could the NovaNET program help students at your school? 



7. Is the program easy to use and does it teach you the subject matter? 
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